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ABSTRACT 

There is a national trend toward using teacher teams and 
collaboration to solve various learning and behavioral problems. Teacher 
collaboration is necessary because of teacher shortages, especially in 
special education, and increasing diversity in student needs. Collaboration 
is especially important in rural schools because of the need to share scant 
resources. During two summers, a rural preparation and training program for 
preservice and in-service special educators was conducted. The program 
consisted of three courses--methods of instruction, learning theories in 
special education, and assessment. To fulfill the practicum component, 
students worked with children with academic, social-emotional, or behavioral 
disabilities who were attending a summer school provided by the university. 
One student from each course served on a collaborative-consultative planning 
team. Assessment of the team process was conducted via an observation 
checklist, a self-evaluation survey, and instructor ratings of individual 
performance. For both summers, all teams progressed through the five stages 
of consultation: establishing the team, identifying the problem, generating 
interventions, implementing interventions, and evaluating the effectiveness 
of interventions. Across all three instruments, preparedness and 
participation were high. All but two students left with a good grasp of team 
process. Pre-service teachers were surprised to discover how many strategies, 
behaviors, and skills could be taught in such a short time and how strategies 
that were implemented worked in such a short time. (TD) 
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EVALUATING TEAMING SKILLS 
IN A RURAL UNIVERSITY CLINICAL EXPERIENCE: 
CONTINUATION ACROSS TWO SUMMERS 



Over the past decade, a national shortage of qualified teachers has been well documented in the education 
literature, causing a high number of unprepared and non-certified personnel to provide instruction to students, 
including those with disabilities (USDOE 2000). This shortage impacts rural special education because openings 
may be more difficult to fill in rural districts, causing a number of teachers to serve on emergency or waiver of 
certification status (Boe, Bobbit, & Terhanian, 1998; Lemke, 1994). Faced with this teacher shortage and the need 
for standards-driven school reform, collaboration and consultation among teachers is required for successful 
problem solving and whole school viability (Friend and Cook, 1990). 

Another reason for the growing need for collaboration is the increasing presentation of diversity in student 
needs. There is a national trend toward using teacher teams and collaboration to solve various learning and 
behavioral problems in public schools. This trend continues in importance because students in public schools are 
continually presenting more diverse needs, and no one teacher possesses all the expertise necessary to address the 
many and varied issues that emerge in classrooms (Johnson, Pugach, & Devlin, 1990). 

Collaborative consultation empowers educators, parents, and students. Collaboration has been common in 
schools even before becoming popular on a broad basis because of the historic need in rural areas to share scant 
resources (Martin & Williams, 2000). Therefore the preparation of pre-service and in-service teachers to participate 
in and contribute to team decision-making for students with diverse and exceptional needs is especially imperative 
in rural areas. 

Team collaboration requires redefinition of roles and structures in conjunction with reflection on practice 
(York, Kronberg, & Doyle, 1995). Team development occurs in stages (Joiner, 1993). Each stage adds a dimension 
to the team’s effectiveness (Are we doing the best job possible?), efficacy (Are we doing the best possible job in the 
best ways possible?), and efficiency (Are we doing the best possible job in a timely manner?). 

In addition, at all levels of education, there is a focus on performance indicators of expected learning. With 
the shift from curriculum-based to performance-based measures t^t require knowledge and sldlls, this method of 
evaluating team member performance provides reliable data to indicate that pre-service teachers perform at levels of 
competence required to meet NCATE Standards. The evaluation tools developed for this project will assist 
instructors to sort out who are meeting national criteria, an essential component of program evaluation. 



Program Description 

During the Summers of 2001 and 2002, a rural preparation and training program for special educators was 
conducted with both a pre-service and an in-service focus. A university-based Summer School was provided for 
community children. Included were children with various academic, social-emotional, and/or behavioral disabilities 
who were referred by their parents and teachers. These Summer school experiences provided clinical opportunities 
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for three university special education courses, Methods of Instruction, Learning Theories in Special Education (the 
Behavior Change course), and Assessment. All three of these courses carry a practicum component. To fulfill the 
practica, Methods students planned and implemented instructional activities for small groups of children. Learning 
Theory students conducted functional analysis, planned and in^lemented behavior change programs for individual 
children, and evaluated the effectiveness of the behavior change. Assessment students conducted formal and 
informal assessment of the academic needs of individual children. 

One student firom each of these courses served on a collaborative-consultative planning team. Each child 
had a planning team. Each student had an expected roll to perform on the team in relation to the child. Teams met 
once per week over a five-week period. Results were brought to team and interpreted in team. Problem solving was 
based on the on-going team assessment. These teams were designed to simulate actual in-school Teacher Assistance 
Teams, which were unavailable to university students during summer months. 

Method 

There were three instruments designed for this project serving two functions. One was an observation 
checklist to provide formative assessment of team process. The other two were summative evaluations of team 
member performance: the reflective self-evaluation tool, and the instructor rating of individual performance. These 
instruments were developed through consultation between the two instructors, and their implementation was 
thoroughly rehearsed to increase reliability of inter-observer agreement and minimize observer drift. 

Team Behavior Observation Checklist was the formative instrument used by the two instructors to document team 
progress across the five team meetings. The form contained a fi*equency chart and a checklist. The fi'equency chart 
documented each member’s demonstration of active listening, contributing, disagreeing agreeably, questioning, and 
encouraging others to participate. These firequencies were totaled for individual team members and across behavioral 
factors. Along with the fi*equency chart was a checklist of the five stages of team planning. Team planning stages 
were as follows: 1) establishing a team, (2) problem identification, (3) generating interventions, (4) intervention 
implementation planning, and (5) evaluation of the intervention. The observing instructor circled each of the stages 
demonstrated by the team during observed a team meeting. Note that a team could demonstrate more than one stage 
during any meeting. 

Team Self-evaluation was a summative instrument used by students to reflect on effective team participation and on 
whole team function. This form consisted of six questions. Items 1-5 were answered by individual team members, 
the team then discussed one or more items of their choice in detail, and responses to item 6 summarized team 
discussion of their chosen items fi'om 1-5. 

Individual Summary Evaluation of Team Work was a summative evaluation of the individual team members’ 
performance by the instructor. This five-point. Likert-type scale addressed how individual members contributed to 
the teaming process and the effectiveness of the team effort using the characteristics of prepared contributions, 
interpersonal skills, team planning, team decision-making, and team evaluation of effectiveness. At the end of the 
practica, the instructor evaluated each student once using this summative tool. 

Research Questions 

To demonstrate a model of collaborative pre-teacher teaming experience and describe its evaluation process the 
following questions were explored: 

1 . Did the team’s behavior progress through the stages of consultation: establishing the team, identifying the 
problem, generating interventions, in^lementing interventions, and evaluating the effectiveness of interventions 
implemented? 

2. Was there evidence of preparation, participation, and process iirqjrovement in the Individual Summary 
Evaluation of team work by meniibers? 
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3. Were the team members able to reflect on and make ut least one recommendation for the in^rovement of team 
process? 

4. What were the cross-validating, common themes among instruments: Team Behavior Observation Checklist, 
Team Self Evaluation, and Individual Summary Evaluation of Team Work? 

5. Would a second set of data yield similar results across research questions 1-4? 

6. To what degree did Assessment and Learning Theory students agree in their responses to the Team Self- 
evaluation items 1 through 5? 

7. To what extent did the team self-evaluation ratings agree with instructor ratings of similar items on the 
Individual Summary Evaluations? 

Analysis 

Multiple case sanpling increases validity, and cases may be chosen on the basis of conceptual grounds 
(Miles & Huberman, 1994). Each evaluation instrument presents a case for con^arison, and instructor-posed 
questions serve to bound the cases in terms of what would be examined. Case con^arison across time, that is 
replicating the described teaming process with another cohort of university students, increases construct validity and 
provides a measure of reliability for these three instructor-made instruments. Hence, this study is a replication of and 
extenstion of previoxis work at this rural university (Friedland & Walz, 2002). 

Question 1: The Team Behavior Observation Checklist was used in order to determine the teams’ progress through 
the stages of consultation. The instructor assigned to observe each particular team recorded incidents of 
demonstrated team behaviors related to the consultation stages at a given meeting. In 2001, there were 45 instructor 
observations made over the five-week period. In 2002, there were 27 instructor observations. The data was analyzed 
by counting the frequency of demonstrations of each stage of team planning. 

Question 2: The Individual Summary Evaluation of Team Work was used to determine the individual members’ 
degree of preparedness, participation and reflection on the team’s process. The instructor rated each student at the 
end of the practica experience. A histogram was constructed of the group’s ratings for both overall performance and 
by each characteristic. 

Question 3; The Team Self-evaluation was qualitatively reviewed to determine whether Learning Theory and 
Assessment students offered at least one suggestion for improving team process in retrospect 

Question 4: Qualitative comparison was conducted to determine if there were some cross- validating common 
themes among these three instruments: Team Behavior Observation Checklist, Team Self-evaluation, and Individual 
Summary Evaluation of Team Work? 

Question 5: A second set of data. Summer 2002, was examined in the same ways as for data collected in the 
Summer of 2001 to determine if similar results would be obtained across research questions 1-4? 

Question 6: Assessment and Learning Theory students’ responses to the Team Self-evaluation items 1 through 5 
were qualitatively con^ared for patterns of agreement 

Question 7: Team Self-evaluation Summaries were compared for agreement with instructor ratings of similar items 
on the Individual Summary Evaluations? 



Results 



For both Summers, 2001 and 2002, checklists of stages of team planning frequency count indicated all 
teams progressed through the five stages of consultation. The first week all teams demonstrated stage one 
establishing the team, although in 2002, teams spent much less time on establishing the team than was spent in 2001. 
During the second week, all teams demonstrated problem identification and generating interventions. During week 
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three, several teams demonstrated planning implementation of interventions. Two teams were still engaged in 
planning the implementation of interventions toward the end of Summer 2002 session because evaluation of their 
intervention led to the conclusion that the intervention was not effective to solve the learning problem, resulting in 
revision and redesign. During week four, stages 2, 3, and 4 were demonstrated by most of the teams. During week 
five, all but one team demonstrated evaluation of the intervention. Therefore it was concluded that all teams 
demonstrated progress through die five-stage consultation process. 

For Summer 2001, Individual Summary Evaluation of Team Work ratings revealed that the range of 
performance was 19 to 25 on the 25-point checldist with one outlier earning a rating of 13. This student missed two 
of the five team meetings, thus she lacked the opportunity to demonstrate the expected skills. This indicates that 
most team members performed at or above the 80% level. For each characteristic identified, most students received 
a rating of 4 or 5, which is high on the 5 point Likert-type scale. These results are summarized as follows: Prepared 
Contributions 20/22 students. Interpersonal communications 18/22 students. Planning 21/22, Collaborative decision 
making 20/22, Evaluation of effectiveness 14/22 with six receiving a rating of three. 

For Summer 2002, Individual Summary Evaluation of Team Work ratings revealed that the range of 
performance was 19 to 25 on the 25-point checldist, with one outlier earning a rating of 16. This student missed the 
last team meeting because of confusion about the meeting time. This, however did not affect her rating. Her pattern 
of refusal to initiate in team and her hesitancy to contribute her findings to the team effort brought her ratings down 
considerably. This indicates that most team members performed at or above the 80% level. For each characteristic 
identified, most students received a rating of 4 or 5, which is high on the 5 point Likert-type scale. Results are 
summarized as follows: Prepared Contributions 19/20 students. Interpersonal communications 19/20 students, 
Planning 19/22, Collaborative decision making 20/20, Evaluation of effectiveness 15/20 with four receiving a rating 
of three. 



Results on the Individual Summary Team Evaluation form are strikingly similar across the two data sets. 
For the most part, students came prepared, participated in the team and reflected on the process. Two students did 
not perform at the level of their peers for very different, individual reasons. One did not attend team meetings on a 
consistent basis, and the other demonstrated difficulty with assertiveness in team. The lower rating received by these 
two students confirms the validity of this instruments to discern students not demonstrating teaming skills. The most 
revealing result of comparing the two sets of data on this instrument was lower ratings in the area of Evaluation of 
effectiveness. 

All students made at least one suggestion for team improvement on the Team Self-evaluation form, and 
some made several. For summer 2001, the one process in^rovement suggestion that most team members agreed 
upon was that more time in team wotild have made problem-solving flow more smoothly. Most opted for more or 
longer team meetings. The second theme included suggestions for improved communication, such as having team 
minutes taken, increasing casual conversation away from meeting times, and establishing clearly agreed upon 
objectives. For Summer 2002, there was general agreement that members participated actively and were well 
prepared. They echoed concerns about not having enough meeting and planning time. Some interesting qualitative 
themes emerged firom responses of this cohort to item 4, which asked what they learned firom teaming. The most 
prominent of these were: 

1) Assertiveness in team ~ “Teaming allows perspectives to be shared that otherwise would not be seen. Therefore, 
information must be shared, so don’t be afiraid to speak up;.” and (2) “The power of observation as a tool for 
change;** the meaning of unobtrusive observation, and then the in^ortance of con^aring observations among 
observers. Several students commented about how well teaming really works to make academics and behaviors 
better, even in the short time afforded by the six weeks of Summer School program. One student expressed surprise 
at how much academic performance influences behavior. 

In comparing Assessment and Learning Theory students* Team Self-evaluation Summary items 1-5, some 
patterns emerged. Even though overall, team members participated and were prepared, it was noted that the two 
Assessment students felt that two of the Learning Theory students were holding back, not contributing what they 
knew to the team effort, “. . .didn’t have much to say.” Conversely, the Learning Theory students felt that, by the 
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time all the assessment information was presented, they did not have adequate time to present behavioral concerns. 
Thus, they felt less supported by the team in their decision-making. Overall, students felt that in order for teaming to 
produce effective planning and successful interventions, flexibility and organization are crucial. 

Instructor evaluation ratings of unique contributions to the team agreed with team self-evaluations in the 
areas of participation and preparedness, with the exception of three individual students. Instructor ratings of team 
efforts differed from team self-evaluations in the area of evaluation of effectiveness for four students. 

Discussion 

Overall, the students realized that it takes more than one person to effectively manage an instructional 
environment. All but two students, one in each Summer session, left with a good grasp of team process. For pre- 
service teachers, the whole experience was an *"eye opener** regarding academic skills deficits and behavioral issues. 
Many were surprised to discover how strategies that were inplemented really worked in such a short period of time, 
and to discover how many strategies, behaviors, and skills could be taught in such a short period of time. 

Similar results were obtained for two Summers in a row. Across the three instruments, preparedness and 
participation were high. Time for teaming and effective communication of children's assessment results and 
behavior analyses presented difficulty. Summer 2002 team members were able to reflect back on the stages of 
teaming as evidenced by several comments they made on the Team Self-evaluation Summary stating that it is 
important to get to know team members and the roles that each must fill. Another area of difficulty expressed was 
establishing goals. Several students expressed that their teams were hesitant to move in a certain direction. In 
Summer 2001, one instructor actually had to prompt the team to move from problem identification and description 
to planning the intervention. During Summer 2002, the other instmetor had to pron^t a team to keep notes so that 
they would have a starting point for each new discussion. 

Instructor evaluation ratings of unique contributions to the team agreed with team self-evaluations in the 
areas of participation and preparedness, with the exception of three individual students. Two of these had difficulty 
with interpersonal communications, and the other had difficulty developing ideas. Instructor ratings of team efforts 
differed from team self-evaluations in the area of evaluation of effectiveness for four students. This area involves the 
ability to reflect in action, or think on one’s feet, choose and implement alternatives, and celebrate success. One 
student gave such short answers that there was no evidence to demonstrate reflection. One had also received a lower 
rating in participation based on her low contribution to team, so there was little basis for reflection. And two simply 
reiterated team comments without much interpretation. 

It may be that this culminating reflective activity is too much all at once at the end of the practicum 
experience. Perhaps teams would benefit more by integrating reflective tasks into their teaming at the end of each 
teani meeting. Perhaps then, Evaluation of effectiveness could more comprehensively address the three Es, 
“Effectiveness, Efficacy, and Efficiency.” 

Implications for Rural Practice 

In these simulated team experiences, students demonstrated evidence of moving through the stages of 
consultative team process. It would follow that, even in a quasi-official learning environment, simulations are 
worthwhile practice to develop teaming and collaboration skills. Therefore, rural teacher preparation programs 
do not have to be limited by lack of local resources or rely on resources from a distance. They can create valid 
experiences in simulated team situations focused on problem solving for children. 



Implications for Further Research 

Action research is a viable method of inquiry that enhances program evaluation and leads to improved 
methods of preparing teachers to meet both the standards of expected performance and the needs of individual 
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children. It is also beneficial for students to see their instructors engage in action research, as this provided a model 
to guide their development of inquiry. 

Several questions that would further verify commonality and validity among instruments remain in process 
of analysis in relation to this study. One such inquiry is to examine the three instruments regarding the students* use 
of reflective techniques in relation to evaluating their teams’ effectiveness, efficacy, and efficiency. What was the 
make up of their on-going reflection? What alternatives were discussed? Were they able to identify needs and 
celebrate their successes? 



References 

Boe, E. E., Cook, L. H., Bobbitt, S. A., & Terhaman, G. (1998). The shortage of fully certified teachers in special 
and regular education. Teacher Education and Special Education, 22, 188-192. 

Friend, M. & Cook, L. (1990). Collaboration as a predictor for success in school reform. Journal of Educational and 
Psychological Consultation, 1(1)^ 69-86. 

Friedland, B. L. & Walz, L. M. (2002, March 8). Evaluating the development of teaming in a rural clinical 

experience using three instructor-designed instruments. 2002 Annual National Conference of the American 
Council on Rural Special Education (ACRES) held in Reno, Nevada, March 7-9. 

Johnson, L. J., Pugach, M. C, & Devlin, S. (1990) Professional collaboration. Teaching Exceptional Children, 22 
(2f 9-11. 

Lemke, J. C. (1994). Teacher induction in rural and small school districts. In D. Montgomery (Ed.), Rural 

partnerships: Working together. Proceedings of the Aimual National Conference of the American Council 
on Rural Special Education (ACRES) held in Austin Texas. March 23-26. (ERIC DRS#ED 369-586). 

Martin, S. M. & Williams, J. M. (2000). Personnel preparation and service delivery issues in rural and remote areas: 
Best practice information. Proceedings of the Annual National Conference of the American Council on 
Rural Special Education (ACRES) held in Alexandria, VA, March 16-18. 

Miles, M. B. & Huberman, A. M. (1994). Qualitative Data Analysis. Thousand Oaks, CA: Sage Publications. 

Scholtes, P. R. (1988). The team handbook: How to use teams to improve quality. Madison, WI: Joiner Associates, 
Inc. 

United States Department of Education (2000). To assure the free and appropriate public education of all children 
with disabilities. Twenty-second annual report to Congress on the implementation of the Individuals with 
Disabilities Education Act (IDEA). Washington, DC. Audiors. 

York, J., Kronberg, R. M., & Doyle, M. B. (1995). Module 4 - Collaboration: Redefining roles, practices, and 
structures. Baltimore, MD: Paul H. Brookes. 



BEST COPY AVAILABLE 

190 

o 

ERIC 



7 



05/13/03 11:00 FAX 435 797 3572 



USU-SPEC EDUC & REHAB 



(S002 



U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 
National Library of Education (NLE) 

Educational Resources Information Center (ERIC) 

ERIC REPRODUCTION RELEASE 

I. Document Identification: 

Title: American Council on Rural Special Education 
2003 Conference Proceedings 
Rural Survival 

March 20-22, 2003; Salt Lake City, Utah 
Author: Multiple - Editor: Ronda Menlove 



Corporate Source: American Council on Rural Special Education (ACRES) 
Publication Date: March 2003 



II. Reproduction Release; 



1^^ oi*der to disseminate as widely as possible timely and signif icantr materials 
of interest to the educational community, documents announced in the? monthly 
abstract journal of the ERIC system. Resources in Education (RIE) , ajre usually 
made available to users in microfiche, reproduced paper copy, and eljectronic 
media, and sold through the ERIC Document Reproduction Service (EDRSl) . Credit 
is given to the source of each document, and, if reproduction releasje is 
granted, one of the following notices is affixed to the document. 



If permission is granted to reproduce and disseminate the identified! document, 
please check one of the following three options and sign the release^ form. 




Level 1 - Permitting reproduction and dissemination in microfiche or other 
ERIC archival media (e.g. electronic) and paper copy. [ 



Level 2A - Permitting reproduction and dissemination in microfiche and in 

electronic media for ERIC archival collection subscribers only. 

I 

Level 2B - Permitting reproduction and dissemination in microfiphe only. 

Documents will be processed as indicated provided reproduction quality permits. 
If permission to reproduce is granted, but no option is marked, documents will 
be processed at Level 1. 



HorB; ** I hereby grant to the Educational Resources Information Center 
(ERIC) nonexclusive permission to reproduce and disseminate this dociiment as 
indicated above. Reproduction from the ERIC microfiche or electronic! media by 
persons other than ERIC ei^loyees and its system contractors require^ 
permission from the copyright holder. Exception is made for non-prof S.t 
reproduction by libraries and other service agencies to satisfy infojrmation 
needs of educators in response to discrete inquiries.” 



Signature 
Printed Name 






Ronda Menlove, Ph.D. 



Position: Proceedings E 
Organization: ACRES 



di-tor 



Telephone No: 435-797-390.1 
Date: 13 May 2003 



Address: 2865 Old Main Hill 
Logan, Utah 84322-2865 



05/13/03 11:01 FAX 435 797 3572 



USU-SPEC EDUC & REHAB 



@003 



III. Document Availability Information (from Non-ERIC Source) j 

If permission to reproduce is not granted to ERIC, or, if you,’ wish ERIC 
o cite the availability of the document from another source, Jplease 
provide the following information regarding the availability df the 
document. (ERIC will not announce a document unless it is publicly 
available, and a dependable source can be specified. Contributors should 
also be aware that ERIC selection criteria are significantly lore 
stringent for documents that cannot be made available through ^ EDRS . ) 

Publisher/Distributor : 



Address : 



Price per copy: 



f 

Quantity price: ' 



IV. Referral of ERIC to Copyright/Reproduction Rights Holder: 

If the right to grant this reproduction release is held by sonieone other 
than the addressee, please complete the following: 

Name : ' 

i 

Address : i 

V. Attach this form to the document being submitted and send both to: 



Velma Mitchell, Acquisitions Coordinator 

ERIC Clearinghouse on Rural Education and Small Schools 

P.O. Box 1348 

1031 Quarrier Street 

Charleston, WV 25325-1348 

Phone and electronic mail numbers: 

800-624-9120 (Clearinghouse toll-free number) 
304-347-0467 (Clearinghouse FAX number) 
mitchelv@ael.org 



